Benefiting from the intrinsic supervision information exploitation capability, contrastive learning has achieved promising performance in the field of deep graph clustering recently. However, we observe that two drawbacks of the positive and negative sample construction mechanisms limit the performance of existing algorithms from further improvement. 1) The quality of positive samples heavily depends on the carefully designed data augmentations, while inappropriate data augmentations would easily lead to the semantic drift and indiscriminative positive samples. 2) The constructed negative samples are not reliable for ignoring important clustering information. To solve these problems, we propose a Cluster-guided Contrastive deep Graph Clustering network (CCGC) by mining the intrinsic supervision information in the high-confidence clustering results. Specifically, instead of conducting complex node or edge perturbation, we construct two views of the graph by designing special Siamese encoders whose weights are not shared between the sibling sub-networks. Then, guided by the high-confidence clustering information, we carefully select and construct the positive samples from the same high-confidence cluster in two views. Moreover, to construct semantic meaningful negative sample pairs, we regard the centers of different high-confidence clusters as negative samples, thus improving the discriminative capability and reliability of the constructed sample pairs. Lastly, we design an objective function to pull close the samples from the same cluster while pushing away those from other clusters by maximizing and minimizing the cross-view cosine similarity between positive and negative samples. Extensive experimental results on six datasets demonstrate the effectiveness of CCGC compared with the existing state-of-the-art algorithms.

Aligning users across networks using graph representation learning has been found effective where the alignment is accomplished in a low-dimensional embedding space. Yet, achieving highly precise alignment is still challenging, especially when nodes with long-range connectivity to the labeled anchors are encountered. To alleviate this limitation, we purposefully designed WL-Align which adopts a regularized representation learning framework to learn distinctive node representations. It extends the Weisfeiler-Lehman Isormorphism Test and learns the alignment in alternating phases of "across-network Weisfeiler-Lehman relabeling" and "proximity-preserving representation learning". The across-network Weisfeiler-Lehman relabeling is achieved through iterating the anchor-based label propagation and a similarity-based hashing to exploit the known anchors' connectivity to different nodes in an efficient and robust manner. The representation learning module preserves the second-order proximity within individual networks and is regularized by the across-network Weisfeiler-Lehman hash labels. Extensive experiments on real-world and synthetic datasets have demonstrated that our proposed WL-Align outperforms the state-of-the-art methods, achieving significant performance improvements in the "exact matching" scenario. Data and code of WL-Align are available at https://github.com/ChenPengGang/WLAlignCode.

Morality in dialogue systems has raised great attention in research recently. A moral dialogue system could better connect users and enhance conversation engagement by gaining users' trust. In this paper, we propose a framework, MoralDial to train and evaluate moral dialogue systems. In our framework, we first explore the communication mechanisms of morality and resolve expressed morality into four sub-modules. The sub-modules indicate the roadmap for building a moral dialogue system. Based on that, we design a simple yet effective method: constructing moral discussions from Rules of Thumb (RoTs) between simulated specific users and the dialogue system. The constructed discussion consists of expressing, explaining, and revising the moral views in dialogue exchanges, which makes conversational models learn morality well in a natural manner. Furthermore, we propose a novel evaluation method in the framework. We evaluate the multiple aspects of morality by judging the relation between dialogue responses and RoTs in discussions, where the multifaceted nature of morality is particularly considered. Automatic and manual experiments demonstrate that our framework is promising to train and evaluate moral dialogue systems.

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding~(VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that our Wukong-Reader has superior performance on various VDU tasks such as information extraction. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.

Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog generation without relying on extra knowledge annotation. Specifically, we use a pre-trained language model to extract the most uncertain tokens in the dialog as keywords. With these keywords, we construct two kinds of knowledge and pre-train a knowledge-grounded response generation model, aiming at handling two different scenarios: (1) the knowledge should be faithfully grounded; (2) it can be selectively used. For the former, the grounding knowledge consists of keywords extracted from the response. For the latter, the grounding knowledge is additionally augmented with keywords extracted from other utterances in the same dialog. Since the knowledge is extracted from the dialog itself, KPT can be easily performed on a large volume and variety of dialogue data. We considered three data sources (open-domain, task-oriented, conversational QA) with a total of 2.5M dialogues. We conduct extensive experiments on various few-shot knowledge-grounded generation tasks, including grounding on dialog acts, knowledge graphs, persona descriptions, and Wikipedia passages. Our comprehensive experiments and analyses demonstrate that KPT consistently outperforms state-of-the-art methods on these tasks with diverse grounding knowledge.

Conceptual knowledge is fundamental to human cognition and knowledge bases. However, existing knowledge probing works only focus on evaluating factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge. Since conceptual knowledge often appears as implicit commonsense behind texts, designing probes for conceptual knowledge is hard. Inspired by knowledge representation schemata, we comprehensively evaluate conceptual knowledge of PLMs by designing three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts, respectively. For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark. Extensive experiments on different sizes and types of PLMs show that existing PLMs systematically lack conceptual knowledge and suffer from various spurious correlations. We believe this is a critical bottleneck for realizing human-like cognition in PLMs. COPEN and our codes are publicly released at https://github.com/THU-KEG/COPEN.

基准数据集在评估自然语言理解（NLU）模型中起重要作用。但是，快捷方式（基准数据集中的不需要的偏差）可能会损害基准数据集在揭示模型的实际功能中的有效性。由于快捷方式在覆盖范围，生产率和语义含义上有所不同，因此NLU专家在创建基准数据集时系统地理解和避免它们是一项挑战。在本文中，我们开发了一个视觉分析系统，即短路，以帮助NLU专家探索NLU基准数据集中的快捷方式。该系统允许用户对快捷方式进行多层次探索。具体而言，统计信息视图可帮助用户掌握统计数据，例如基准数据集中快捷方式的覆盖范围和生产率。模板视图采用层次和可解释的模板来汇总不同类型的快捷方式。实例视图允许用户检查快捷方式涵盖的相应实例。我们进行案例研究和专家访谈，以评估系统的有效性和可用性。结果表明，饭店支持用户通过快捷方式更好地了解基准数据集问题，从而激发他们创建具有挑战性和相关的基准数据集。

我们提出了Pangu-Coder，这是一种仅预读的解码器语言模型，该模型采用pangu-alpha架构进行文本到代码生成，即给定自然语言问题描述的编程语言解决方案的合成。我们使用两阶段策略训练Pangu-Coder：第一阶段采用因果语言建模（CLM）来预先培训原始编程语言数据，而第二阶段则使用因果语言建模和掩盖语言建模（MLM）的组合培训目标，专注于文本到代码生成的下游任务，并培训松散的自然语言程序定义和代码功能。最后，我们讨论了pangu-coder-ft，该pander the是通过竞争性编程问题和代码与持续集成测试的结合进行了微调的。我们评估了pangu-coder，重点是它是否生成功能上正确的程序，并证明它在参加较小的上下文窗口和较少的数据培训的同时，它比诸如Codex之类的类似大小的模型（例如Codex）实现等效性或更好的性能。

B扫描超声模式中图像的精确和快速分类对于诊断眼部疾病至关重要。然而，在超声波中区分各种疾病仍然挑战经验丰富的眼科医生。因此，在这项工作中开发了一个新颖的对比度截面网络（CDNET），旨在应对超声图像中眼异常的细粒度图像分类（FGIC）挑战，包括眼内肿瘤（IOT），视网膜脱离（RD），后堆肥葡萄球菌（PSS）和玻璃体出血（VH）。 CDNET的三个基本组成部分分别是弱监督的病变定位模块（WSLL），对比度多Zoom（CMZ）策略和超级性对比度分解损失（HCD-LOSS）。这些组件促进了在输入和输出方面的细粒度识别的特征分离。所提出的CDNET在我们的ZJU Ocular Ultrasound数据集（Zjuuld）上进行了验证，该数据集由5213个样品组成。此外，在两个公共且广泛使用的胸部X射线FGIC基准上验证了CDNET的概括能力。定量和定性结果证明了我们提出的CDNET的功效，该CDNET在FGIC任务中实现了最新的性能。代码可在以下网址获得：https：//github.com/zeroonegame/cdnet-for-ous-fgic。

在本文中，我们介绍了基于大型预训练的语言模型（PLM）pangu-alpha（Zeng等，2021）的中国预训练的开放域对话生成模型。与其他对大量对话数据进行培训的预训练的对话模型不同，我们旨在通过继承PLM的有价值的语言能力和知识来构建强大的对话模型，并以相对较少的数据和计算成本构建强大的对话模型。为此，我们训练大型PLM Pangu-Alpha的Pangu-bot，该机器人已被证明在各种中国自然语言任务上表现出色。我们研究了pangu-bot产生的响应的不同方面，包括响应质量，知识和安全性。我们表明，Pangu-Bot优于最先进的中国对话系统（CDIALGPT（Wang等，2020），Eva（Zhou等，2021），EVA2.0（Gu等，2022）） W.R.T.以上三个方面。我们还证明，可以轻松地部署pangu-bot，以在没有进一步训练的情况下产生情感反应。在整个经验分析中，我们还指出，Pangu-bot响应质量，知识正确性和安全性仍然远非完美，进一步的探索对于建立可靠且智能的对话系统是必不可少的。我们的型号和代码将在https://github.com/huawei-noah/pretretaining-language-model/tree/master/master/pangu-bot上提供。